IEEE Access 


Multidisciplinary } Rapid Review : Open Access Journal 


Digital Object Identifier 10.1109/ACCESS.2022.Doi Number 


An Approach for Full Reinforcement-based 
Biometric Score Fusion 


Mohamed Cheniti', Zahid Akhtar?, (Senior Member, IEEE), Chandranath Adak?, (Senior 
Member, IEEE), and Kamran Siddique’, (Senior Member, IEEE) 


'Faculty of Electrical Engineering, Electronics department, University of Science and Technology Houari Boumediene, BP.32, EI-Alia, 16111 Bab- 
Ezzouar, Algiers, Algeria 


*Department of Network and Computer Security, State University of New York Polytechnic Institute, Utica, NY 13502, USA 
Department of Computer Science and Engineering, Indian Institute of Technology Patna, Bihar, 801106, India. 
‘Department of Computer Science and Engineering, University of Alaska Anchorage, Anchorage, AK 99508, USA 


Corresponding authors: Mohamed Cheniti (mohamed.cheniti @usthb.edu.dz) and Zahid Akhtar (akhtarz @sunypoly.edu). 


ABSTRACT Multibiometric systems have the potential to mitigate error rates and address certain inherent 
weaknesses found in unimodal systems. This study introduces an innovative scheme for user recognition in 
multibiometric systems, centered on a score-level fusion framework. The foundation of this framework lies 
in the full reinforcement operator (FRO), specifically estimating FRO through generator functions associated 
with triangular norms (T-norms and T-conorm). The efficiency of the proposed method has been showcased 
through an extensive set of experiments carried out on four commonly available benchmark databases: all 
three partitions of the National Institute of Standards and Technology (NIST) databases (Set 1, 2, 3), along 
with the XM2VTS database. Our method achieves superior accuracy compared to existing methods, reaching 
100 % recognition on NIST-Set 1, 93.40 % on NIST-Set 2, and 94.54 % on the more challenging NIST-Set 
3. The experimental findings illustrate that score fusion schemes based on FRO not only enhance verification 
rates when compared to current score-level fusion techniques (such as Asymmetric Aggregation Operators, 
Minimum, Maximum, T-norms, and Symmetric-Sum) but also offer a swift computational performance. 


INDEX TERMS Unidiomatic system, multibiometric system, score-level fusion, full reinforcement operator 


(FRO). 


I. INTRODUCTION 

The established methods of authentication are currently 
facing substantial competition from emerging alternatives. 
Novel technologies like biometric and multibiometric 
systems pose challenges to conventional recognition 
methods. This shift is primarily driven by the significant 
increase in identity theft and security-related concerns. 
These technologies rely on unique human attributes, either 
anatomical (e.g., facial features, fingerprints, iris patterns) 
or behavioral (e.g., cognitive biometrics, gait, keystroke 
dynamics) traits [1]. Furthermore, biometric modalities are 
inherently present in individuals, making them 
exceptionally challenging to counterfeit or pilfer [2]. 
Besides, biometric characteristics establish a robust 
connection between a user and their identity [3]. 
Furthermore, a system that relies on human attributes is 
regarded as a better choice in terms of dependability and 
accuracy [4]. 


Many researchers contend that unimodal biometric systems, 
which rely on a single trait, are not sufficiently equipped to 
tackle challenges such as noisy input data, the limited 
distinctiveness of the chosen biometric trait, non-universality, 
low interoperability, and vulnerability to spoofing [5]. These 
limitations can result in reduced accuracy [5]. Some of the 
shortcomings associated with unimodal systems can be 
mitigated by adopting multibiometric systems, which 
incorporate data from several biometric modalities. Utilizing 
at least two biometric traits offers several advantages 
compared to relying on a single modality [6]. Several 
advantages arise from this approach, including enhancements 
in overall accuracy, the effective management of universality 
concerns, and heightened resistance to spoofing attempts. By 
combining evidence from multiple biometric sources, it 
becomes significantly more challenging for impostors to 
simultaneously imitate various physiological and/or 
behavioral attributes of a genuine user [7]. According to the 


This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and 
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2024.3384544 


IEEE Access 


Mubhtidisciplinary : Rapid Review ; Open Access Journal 


nature of the biometric information source, a multibiometric 
recognition can be categorized broadly into several types, such 
as multimodal (utilizing multiple biometric traits) [8], multi- 
unit (employing multiple instances of the same kind of 
biometric data, e.g., both left and right wrist veins in humans) 
[8], multi-algorithm (applying diverse feature extraction 
methods on the identical biometric characteristic) [8], multi- 
sensor (utilizing multiple sensors for capturing the identical 
biometric trait) [9], and multi-sample (gathering a multitude 
of samples of the identical biometric characteristic) [9]. To 
address the limitations observed in prior research on both 
unimodal and multimodal biometrics, it is imperative to 
develop contemporary and computationally efficient methods 
for combining authentication scores. 

In this article, we have introduced an innovative approach 
for integrating biometric scores at the score-level, known as 
the “full reinforcement operator”. This operator is employed 
to amalgamate scores generated from various sources, and 
compared to other score fusion techniques like Dempster- 
Shafer (DS) theory, symmetric-sum, T-norms, weighted sum, 
and max rules, it shows improved differentiation between 
genuine and imposter match-scores. Our suggested 
multibiometric score fusion framework, based on the full 
reinforcement operator, was rigorously validated in diverse 
user authentication scenarios, including multiple traits 
(utilized both the XM2VTS database [31] and the NIST- 
multimodal database [7]), multiple instances (using the NIST 
fingerprint database), and multiple algorithms or features 
(merging output scores from two separate face matchers, 
NIST face database). 

The rest of our paper is organized as follows. Section II 
delves into the relevant literature concerning the combination 
of matching scores. In Section HI, we offer a detailed 
explanation about reinforcement in aggregation operators. 
Section IV presents a concise description of the proposed 
score fusion technique. Section V is dedicated to discussing 
the experimental results we have obtained. Finally, Section VI 
concludes this paper. 


ll. Related Work 

Information fusion contributes significantly in multimodal 
biometric systems, the key factor behind the success of 
multimodal biometric is fully related on how we select the 
fusion schemes to enhance performance. In literature, 
matching score-level is generally used even though there are 
many levels of fusion (e.g., decision, features). Moreover, 
there is a categorization of matching score fusion techniques, 
which falls into three main groups: classifier-based, density- 
based, and transformation-based score fusion. Within the 
transformation-based category, these techniques are broadly 
separated into a pair of subgroups: fixed and trained score 
fusion rules. Before implementing either fixed or trained rule- 
based fusion methods, employing normalization techniques 
like tanh, or double sigmoid, min-max scaling, z-score 
standardization (normalization), is crucial to first convert the 
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matched scores into a common domain and to ensure a 
meaningful integration. Fixed rules, such as symmetric sum, 
product T-norm, minimum, maximum, and Asym-AOs 
(Asymmetric Aggregation Operators), do not require any 
specific learning process. In contrast, trained rules necessitate 
a learning phase, as seen in techniques like weighted sum. 


A. Fixed rule-based score fusion 

Following the normalization process, the scores obtained from 
each matcher are amalgamated using a designated rule, such 
as minimum, maximum, or multiplication. 

For instance, in 2013, Wang et al. [10], created a multimodal 
biometric system that could recognize faces and irises. To 
combine the results, they used a score-level fusion method 
based on the Aczél-Alsina triangular norm (T-norm). The 
CASIA-Iris-Thousand database, which has dual iris data, and 
the NVIE face database, which contains both visible and 
thermal facial photos, are used to test the experiment's vertical 
databases. Their fusion rule has EER (equal error rate) = 
2.98.10~*, which represents an important improvement than 
unimodal system. In that very year, Vishi et al. [11], 
introduced an innovative method for multimodal biometric 
verification that combines iris and fingerprint characteristics 
at the score-level. They employed various normalization 
techniques, including Min-Max, Z-Score, and Hyperbolic 
Tangent, along with three predefined score fusion methods: 
Minimum Score, Maximum Score, and Simple Sum. They 
conducted fusion experiments across four databases, utilizing 
FP-DB1 and FP-DB2 for fingerprint data and Iris-DB1 and 
Iris-DB2 for iris data. The authors found that employing the 
hyperbolic tangent estimator for score normalization and 
applying the simple sum rule fusion resulted in an Equal Error 
Rate (EER) of 0.0001 % for the combined fingerprint and iris 
data. On one hand, better than the unimodal biometric system 
and on the other hand better than both max and min score 
fusion rules. 

Moeen et al. [12] introduced a score-level fusion technique 
for person recognition, utilizing the entropy function 
(Hanman-Anirban). For experimentation, they used the finger- 
knuckle-print images obtained from (FKP) dataset consisting 
of left index, left middle, right index and right middle FKP. 
The findings demonstrate that the suggested method achieves 
a notable enhancement in performance when compared to 
individual FKP approaches. They obtained genuine 
acceptance rate of 99 % with FAR of 0.001 %. 

Cheniti et al. [7] proposed score-level fusion of multimodal 
biometrics using the symmetric sum rule and used T-norm to 
generate the symmetric sum functions. The provided 
framework underwent testing on two openly accessible 
benchmark databases. Specifically, the authors utilized a pair 
segment of NIST-BSSRI, namely, the NIST- fingerprint 
database and the NIST-multimodal database. The empirical 
outcomes demonstrate that the suggested technique surpasses 
the performance of current methods when applied to both the 
NIST-multimodal database and the NIST fingerprint database. 
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In 2020, Herbadji et al. [8] developed score fusion technique 
for person authentication using weighted quasi-arithmetic 
mean. The authors conducted experiments using three publicly 
accessible datasets: NIST-BSSR1 Fingerprint, NIST-BSSR1 
Multimodal, and NIST-BSSR1 Face. In their evaluation of 
multimodal, multi-unit, and multi-algorithm systems, they 
demonstrated that the WQAM combining rule they proposed 
surpasses the earlier suggested score combining method, 
which relied on fixed techniques like T-norms, trained 
methods such as support vector machines, and density 
assessment approaches like likelihood ratio. Table I represents 
a summary of the contemporary fixed rule-based score fusion. 


TABLE I: SUMMARY OF CONTEMPORARY FIXED RULE-BASED SCORE 
FUSION 
Biometric Traits 


Method Year Fusion Rule 


[32] 2012 Adaptive weighted Palmprint 
[10] 2013 Aczél-Alsina iris and thermal face 
T-norm images 
[11] 2013 Sum, max, min rule iris and fingerprint 
[34] 2013 Mean, acces Palm-print-vein 
median rule 
[13] 2014 Mean rule face and fingerprint 
[14] 2015 Max tule palm-dorsal vein 
: finger-knuckle-print (left 
[12] 2016 See asad index, left middle, right 
PY index and right middle) 
[33] 2017. Adaptive weighted Left and right palmprint 
[15] 2017 Dubois and Parad left and right wrist vein 
T-norm patterns 
[7] 2018 Symmetricsumrule face and fingerprint 
[16] 2019 T-norm wrist, palm vein images 
Weighted quasi- : ; : 
[8] 2020 adihiene Hea face and fingerprint 
(9] 2020 Asym-AO palmprint, fingerprint, 


face and ocular left eye 


B. Trained rule-based score fusion 

Prior to employing the trained rule, it is essential to split the 
matcher scores into two distinct sets. One of these sets is 
allocated for the training phase, while the other is reserved for 
the testing phase. The process of training the rule involves 
learning and determining the appropriate model parameters, 
such as identifying the correct weights in the case of weighted 
sum or product rules. These learned parameters are then 
applied in the subsequent testing phase. 

One of the representative works of trained rules is [17]. 
Furthermore, in [18], the authors introduced a score fusion 
technique that integrates belief functions for palmprint, face 
and iris modalities. The authors employed Gabor filter 
combined with dimensionality reduction techniques PCA, 
LDA and KFA to transform matching scores into belief 
assignments, and they utilized particle swarm optimization 
(PSO) to determine the weights assigned to the belief 
assignments of the palmprint, face and iris classifiers. 
Subsequently, the Dempster-Shafer (DS) theory was applied 
to combine the masses. While the authors [19] devised an 
effective score fusion method utilizing the Dezert- 
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Smarandache theory (DSmT). The suggested score fusion 
framework consists of three main steps: (i) calculation of the 
generalized basic belief assignment, (ii) fusion of the assigned 
beliefs using a fusion rule based on DSmT, and (iii) 
determination of whether to accept or reject a user. A summary 
of some representative studies in trained rule-based score- 
level fusion systems is presented in Table II. 


TABLE II: SUMMARY OF CONTEMPORARY TRAINED RULE-BASED SCORE 
FUSION 
Method Year 


Fusion Rule Biometric Traits 


19] 2015  weighted-sum rule Voice and face 

18] 2019 PSO-DS Palmprint, iris, face 

20] 2015  weighted-sum rule Face and iris 

21] 2016 weighted-sum rule Face and iris 

22] 2017 weighted-sum rule Face and iris 

23] 2017 Quality-based weights — Voice and face 

24] 2018 DSmT-based rule Face and Fingerprint 

25] 2019  Backtracking Search, Tris, finger-vein and 
PCR fingerprint 

26] 2020 weighted-sum rule Finger-vein 


Ill. Boosting Aggregation via Reinforcement Strategies 
Several attempts have been conducted to find the best score- 
level fusion technique; moreover, many researchers have 
claimed that an effective score-level fusion technique should 
aim to maximize genuine scores while minimizing impostor 
scores, ultimately leading to an enhanced genuine 
acceptance rate (GAR) and reduced false acceptance rate 
(FAR). The only fusion rule that satisfies the required 
properties, i.e., simultaneously maximizes the genuine 
scores and minimizes the impostor scores, is called a full 
reinforcement operator (FRO) [27]. 

By employing the FRO fusion rule, we can ensure, firstly, 
that a set of genuine scores work together to strengthen and 
affirm the fused score, surpassing the individual scores 
alone. Secondly, a set of impostor scores collaborate to 
intensify the disconfirming nature of the fused score, 
surpassing the individual scores alone. To delve into more 
specific aspects, we need to introduce two closely related 
methods. The first one is referred to as the upward 
reinforcement operator (URO) [27]. With this approach, we 
are describing the tendency where, if the scores used for user 
identification are all highly favorable (indicating genuine 
scores), they strengthen each other, resulting in even stronger 
confirmation that the user falls under the genuine category. 
Likewise, the downward reinforcement operator (DRO) 
represents the inclination where, if the scores used for user 
identification are all low (indicating impostor scores), they 
strengthen each other, leading to a more pronounced 
confirmation that the user belongs to the impostor category 
[27]. Overall, a score fusion rule that displays both URO and 
DRO is called a full reinforcement operator (FRO) [27]. 

A score fusion rule W, operating with input scores taken 
from the unit interval [0,1], is considered to exhibit upward 
reinforcement exclusively, when its output is greater than the 
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maximum of its individual arguments (input scores). 
Similarly, a score fusion rule W, is said to display downward 
reinforcement only when its output is lower than the 
minimum of its individual arguments. 


Example 1. Assume a multimodal biometric system is 
characterized by three score matchers M7, M2, M3. Let W 
be a fusion rule. If for the user x we have: M/ (x) = 0.8, M2 
(x) = 0.85, M3(x) = 0.9 and if W (0.8,0.85,0.9) = 0.91, Then 
W is an upward reinforcement operator (URO). 


Example 2. Assume a multimodal biometric system is 
characterized by three score matchers M1, M2, M3. Let W 
be a fusion rule. If for the user x we have: M/ (x) = 0.4, M2 
(x) = 0.3, M3(x) = 0.35, and if W (0.4,0.3,0.35) = 0.2, Then 
W is a downward reinforcement operator (DRO). 


As full reinforcement operators can offer both attributes 

concurrently, encompassing both the upward and downward 
reinforcement characteristics, they seem to be a desirable 
quality to incorporate in score-level fusion techniques. We 
aim to study if the previously suggested score fusion rules 
can manifest this property. Because there are ample of fusion 
rules in the literature, we prefer to begin with T-norms 
because they confirm their ability to provide a good 
performance [7], and they are adopted in many studies as a 
generator to produce some other fusion rules such as 
symmetric sum [7], Asym-AOs [9]. Additionally, they can 
represent a broad class of several fusion rules such as min, 
mean, max, product and average sum. 
T-norms, often known as triangular norms, are a class of 
binary functions that generalize intersection at fuzzy sets. A 
T-norm is a function T: [0,1] x [0,1] — [0,1] that meets 
the requirements listed below: 


i) T is commutative: T(x, y) = T(y, x). 
ii) Tassociative : T (x,T (y,z) = T (T (,y),Z). 
iii) 1 is an identity operator:T (x,1) = T (1,x) = 1. 
iv) Tis rising in each variable: 

ysz=>T@y) <T@,2z). 
An indispensable property associated with any T-norm is: 

T (X41, -5%p) < min,[x;] 

From this last property we can say that any biometric score 
fusion rule base on T-norm exhibits the property of 
downward reinforcement. Moreover, from the same property 
we can see that any output of T-norm can never exceed the 
minimum of the input scores (arguments), therefore, T-norm 
does not exhibit upward reinforcement. From _ this 
investigation we can conclude that T-norm based score-level 
fusion is very useful when it deals with impostor scores, it 
indeed reinforces them to a lower value, but it fails to 
reinforce the genuine scores, unfortunately the T-norm is not 
full reinforcement operator [27]. As we said above the T- 
norm is a universal class of numerous score fusion rule like 
symmetric sum, max, min, product, and Asym-AOs. We can 
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conclude that all these rules are not full reinforcement 
operator [27]. 

Let’s shift our focus to the T-conorm aggregation operator, 
as described in references [15]. This operator has been 
employed in the field of fuzzy set studies, serving as an 
extension of disjunction or type aggregation. A T-conorm is 
a function C:[0,1] x [0,1] — [0,1] that satisfies the 
following properties: 


i) C is commutative: C(x, y) = C(y,x). 
ii) C is associative: C (x,C (y,z) = C (C (~,y),z). 
iii) 0 is an identity operator: C (x,0) = C (0,x) = 0. 
iv) C is rising in each variable: 
ysz=C(x%,y) < C(%,z). 
An indispensable property associated with any T-conorm is: 
C(Xq, Xp) = max;[x;] 
From this characteristic indicates that the T-conorm 
functions as an upward reinforcement operator. When all 
scores are high, the combined value will be equal to or 
greater than the highest individual score. However, it is 
worth noting that T-conorm do not function as downward 
reinforcement operators. In situations where all inputs are 
low, the result does not fall below the minimum value but 
instead equals or exceeds the maximum value among the 
aggregated scores. 
Now we see if the mean fusion-rule exhibits the full 
reinforcement property, we select mean rule-based score 
fusion for the examination because it represents a general 
aggregation class to many other score fusion rules such as 
medium, arithmetic, weighted sum, ordered weighted 
averaging (OWA) [19] and Weighted quasi-arithmetic mean 
[8]. A mean operator Z is a mapping Z: 1" — I (I is the unit 
interval), such that it is: 
i) Commutative. 
i1) Monotonic. 
iii) Idempotent Z(x1,...,%n) = Xx. 
A substantial and understandable property associated with 
any kind of mean operator score fusion rule is that its output 
is always between the max and min of its inputs, 
Le., min(x,) < Z(%,..,X%,) < max;(x%}) 
Due to the aforementioned characteristic, a mean operator 
score fusion cannot be a URO or DRO, and consequently 
neither the mean rule nor its subclasses are a FRO [27]. After 
the examination of three general classes of score fusion rule 
and their subclasses, we can conclude that all previous fusion 
score-level fusion rules are not optimized because they fail 
to exhibit the full reinforcement property. This fact motivates 
us to introduce a score-level fusion rule based on FRO. 


IV. Proposed Full Reinforcement Score Fusion 
Technique 

In this section, we define an innovative technique for score- 

level fusion that leverages two interconnected methods: 

upward and downward reinforcement. Initially, we explore 

the fuzzy models associated with the full reinforcement 
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approach, followed by an explanation of the multibiometric 
score fusion method based on FRO. 


A. Reinforcement from fuzzy models 

The above investigation led us to conclude that the majority 
of score fusion rules previously proposed in the literature do 
not possess the capabilities of a full reinforcement operator, 
which motivated us to adopt a score fusion rule based on a 
full reinforcement operator. 

Fuzzy logic theory presents a robust framework for 
knowledge representation within the domain of intelligent 
system construction. This framework proves particularly 
advantageous in a diverse range of applications, including 
pattern recognition, information retrieval, diagnostic 
systems, and multi-criteria decision-making [35]. For this 
reason, we adopt a similar fuzzy logic modeling-based 
technique of [27]. Initially, it's essential to recall that a T- 
norm operates as a downward reinforcement operator, while 
a T-conorm serves as an upward reinforcement operator. 
Consequently, we can understand that combining both a T- 
norm and a T-conorm allows us to create the desired 
operator, with the ability to function as a T-norm when 
scores are low and as a T-conorm when scores are high. The 
general form of full reinforcement operator can be 
formulated in following two distinguishing rules [27]: 

Rule 1 (R1): When all scores are low, employ a T-norm 
aggregation. 

Rule 2 (R2): When all scores are high, utilize a T-conorm 
aggregation. 

Based on the two rules, we can develop a fuzzy logic model. 
Suppose our score fusion rule comprises n scores, where 
each score is represented by the variable S;. Additionally, we 
define two fuzzy subsets for each S; on the closed interval: 
L;, representing to the “low” notion, and H;, corresponding 
to the “high” concept. Furthermore, we introduce two 
operators: T, representing any T-norm, and C, representing 
any T-conorm. Let M(S;) be the full reinforcement operator. 
Consequently, we can formulate a fuzzy model with a pair of 
constraints as follows: 

R1: If Sy is L, and S, is Ly .... S, isL, then M(S;) is B, 
R2:1f S, is Hy and Sz is Hz .... S, is H, then M(S;) is By 
In the context mentioned earlier, both B, and By, are 
singleton fuzzy subsets (refer to Eqn. 1, 2), where: 


n= fee) @ 
= bes al @) 


We observe that in the previously described model, the 
outcome value depends on the input factors, resembling a 
Sugeno-Takagi framework. Solving this type of fuzzy 
system model involves four distinct steps [27], as below: 

i) Locate the firing level t;of every constraint (refer to Eqn. 
3, 4). 
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ii) Detect the functional output fuzzy membership function 
of each rule. 

iii) Fuse these functional outputs (refer to Eqn. 5). 

iv) Crispify the resulting fuzzy set (refer to Eqn. 6). 

For rule one the firing level is: 


n 
T= | Juco (3) 
i=1 
Similarly, for rule two we get: 
n 
t= | [mcs (4) 
i=1 
Combining these two rule-outputs, the system produces: 
Ty T2 
F={ (5) 
T (S41, 82, Sn) C(Sy, So, Sp) 


Using the center of area method of crispification, we get: 


Tz T (Sy, So, «0, Sn) + Te C(S4, So, 0,8: 
M(S,) = 1 T(S1,S2 in) 2 C(S1,S2 n) (6) 
Ty + To 
If we denote: 
ioe eas , ‘i es 


Then 


M(Spn) = WyT (Sy, S82, Sn) + Wz C(Sy,S2,4-,Sn) (8) 
Thus, we see that M (refer to Eqn.8), is obtained as a kind of 
weighted average of T-norm and T-conorm. However, it is 
important to emphasize that this approach diverges from a 
basic weighted average involving a T-norm and a T-conorm, 
as we have previously demonstrated that such a method does 
not achieve full reinforcement. Here, the key distinction lies 
in the fact that the weights (refer to Eqn. 7) are not fixed 


constants; instead, they are contingent upon the scores S;. 


B. The score fusion method based on FRO 
The primary purpose of a biometric system is to perform the 
detection of patterns or matching. The two main duties of 
biometric recognition systems are registration and 
verification. In the registration phase, A biometric sensor, 
like a fingerprint reader in the case of fingerprint recognition, 
is used to acquire the biometric attribute. The derived 
biometric feature is then used to extract salient 
characteristics, frequently referred to as a “template”. It is 
then entered along with the name of user in a database. When 
a user requests to be recognized during the recognition or 
verification phase, they give the system a biometric 
characteristic. This stage involves extracting features using 
the biometric trait collected from the sensor, evaluating those 
features to traits in the enrollment database, and generating a 
matching score. If the matching score surpasses a predefined 
threshold, the system classifies the user as genuine; 
otherwise, it identifies them as impostors. 

Figure 1 depicts an architectural representation of the 
entire procedure of a multibiometric person authentication 
scheme. This framework combines data from various 
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biometric sources using Full Reinforcement Aggregation 


Operators (FRO). 


eee 

e@eee 
Score Normalization 
Fusion using FRO 


Mater N 
Modality N 


FIGURE 1. Multibiometric person authentication using FRO. 


Due to the diversity in match scores, it is necessary to 
initially convert these various match scores into a single 
range of [0,1] prior to fusion. In this study, we employed two 
normalization techniques, specifically the min-max and 
tanh-estimators, as described here: 
; S —min(S) 
~ max(S) — min(S) (9) 
Here, S’ denotes the adjusted score, and S signifies the match 
score produced by a particular matcher. 


1 _ 
S' =5 tanh (o01(=*)) “ | (10) 


In this context, S’ represents the adjusted score, where S, 
and o correspond to the input score, mean, and standard 
deviation of input scores, respectively. These values are 
determined using the Hampel estimator [7]. After 
normalizing the match scores using either the min-max 
method (refer to Eqn. 9) or the tanh-estimators (refer to Eqn. 
10), the FRO combines these standardized scores. In this 
paper, a specific instance of FRO is employed, referred to as 
the triple [] combination rule [27]. This operator is created 
by utilizing the product T-norm and its corresponding dual 
for T-conorm (refer to Eqn. 11, 12). 


n 
T =T(S1,S2,+,5n) = | Is (11) 
j=1 
n 
C = C(Sy, So) 0)5n) =1- | [5 (12) 
j=l 


ns. 
ee |) 
jar i + Wa Sj 
where, T, C, and M correspond to the T-norm, T-conorm, 
and triple aggregation operator ([]), respectively. 
If the combined score surpasses a certain threshold (set 
empirically), the user is verified as genuine; otherwise, 
identified as an impostor. 
V. Experiments 
In this section, we present the performed experiments and 
results of the proposed score fusion approach, i.e., Full 
Reinforcement Operator (FRO). More specifically, we have 
performed experiments using four publicly accessible 


M(Sj,5S,.+,Sn) = 
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datasets to evaluate this method’s effectiveness in diverse 
scenarios, including multimodal, multi-unit, and multi- 
algorithm systems, utilizing four publicly available 
databases. 


A. Experiments Data 


The study made use of four databases that are openly 
accessible, including the three sets of NIST BSSRI1 
databases (multimode, face, fingerprint) [28], and the 
XM2VTS database [31]. 


1) Set 1 (NIST-Multimodal) 


Set 1 comprises scores generated from a cohesive group of 
517 individuals, incorporating both face and fingerprint data. 
Each person in this set has one score derived from comparing 
two right index fingerprints, another from comparing two left 
index fingerprints, and two additional scores from 
comparing two facial images. It is important to note that the 
fingerprint and face images used to generate these scores 
belong to the same individual, and were captured 
simultaneously (with documented dates provided for 
reference). 


2) Set 2 (NIST-Fingerprint) 


Set 2 consists of fingerprint scores obtained from a unified 
system applied to images of 6000 individuals. For each 
individual in this dataset, there is one score obtained by 
comparing two left index fingerprints and another score from 
comparing two right index fingerprints. 


3) Set 3 (NIST-Face) 


Set 3 includes scores derived from two facial recognition 
systems applied to images of 3000 individuals. In this 
dataset, each individual is represented by one score obtained 
by comparing face A with a subsequent face, B, and another 
score from comparing face A with a different subsequent 
face, C. 


4) XM2VTS Database 


The XM2VTS Benchmark database is an expanded version 
of the m2vts database and has maintained its reputation as 
one of the most extensive publicly accessible audio-visual 
speech databases since its inception. It offers approximately 
30 hours of unprocessed video footage. In this database, the 
295 speakers are divided into 200 clients and 95 impostors. 
There are two methods for dividing the set used for both 
training and evaluation, known as Lausanne Protocol I and 
II, denoted as LP! and LP2 [29]. The specifics of the 
Lausanne Protocol can be found in Table III and IV. Facial 
data undergoes three different feature extraction methods: 
FH (Face Histogram), DCTs (Discrete Cosine Transform for 
small images), and DCTb (Discrete Cosine Transform for 
large images). For speech data, three distinct feature 
extraction methods are employed: LFCC (Linear Filter-bank 
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Cepstral Coefficient), PAC (Phase Auto Correlation MFCC), 
and SSC (Spectral Sub-band Centroid). 


TABLE III: BRIEF EXPLANATION OF LAUSANNE PROTOCOLS I [29] 


Lausanne Protocol I 


Number Numberof Number 
Set Clients/ of recordings _ of scores 
Impostor subjects per subject 
Training set Clients 200 3 600 
Evaluation Clients 200 3 600 
set Impostor 25 8 40000 
Test set Clients 200 2 400 
Impostor 70 8 11200 


TABLE IV: BRIEF EXPLANATION OF LAUSANNE PROTOCOLS II [29] 


Lausanne Protocol II 


Number Number of | Number 
Set Clients/ of recordings of scores 
Impostor subjects per subject 
Training set Clients 200 4 800 
Evaluation Clients 200 2 400 
set Impostor 25 8 40000 
Test set Clients 200 2, 400 
Impostor 70 8 11200 


B. Experiment Protocol for databases 


For the three sets of NIST BSSRI databases the 
authentication mode experiments were conducted, and the 
effectiveness of the proposed fusion method was assessed 
using Receiver Operating Characteristics (ROC) analysis. 
The ROC curve is generated by plotting the Genuine 
Acceptance Rate (GAR) against the False Acceptance Rate 
(FAR). GAR is defined as (1 — FRR) [7], where FRR 
(False Rejection Rate) represents the percentage of 
legitimate individuals that the system mistakenly rejects as 
impostors, and FAR indicates the percentage of impostor 
individuals that the system incorrectly accepts as client users. 
Meanwhile, GAR measures the rate at which client users are 
correctly accepted out of the total number of enrolled 
individuals. 

For XM2VTS Benchmark database, The Half Total Error 
Rate (HTER) is used to compare the performance of the 
different fusion techniques [29]. It is defined as: 


FAR +FRR 
HTER = — (14) 
It is crucial to highlight that the FAR and FRR do not exhibit 
the same level of sensitivity. This discrepancy arises from 
the presence of more simulated impostor attempts than 
genuine client attempts. Consequently, the FRR experiences 
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more significant fluctuations when erroneously rejecting a 
genuine client access, whereas the FAR exhibits less 
dramatic variations accepting an impostor access mistakenly 
[29]. 


C. Experiments Results 

In this section, we present the experimental outcomes 
obtained from four publicly accessible databases (NIST- 
multimodal, NIST-Fingerprint, NIST-Face and XM2VTS) 
involving multimodal, multi-unit, and multi-algorithm 
biometric systems. 


1) Performance of FRO-based fusion on NIST-Set 1 


Figure 2 presents an architecture illustrating the entire 
process of a multimodal person recognition framework 
utilizing face and fingerprint data. The scores obtained from 
the face matcher C, face matcher G, as well as the left and 
right fingerprint scores, undergo normalization using the 
tanh-estimator normalization technique (refer to Eqn. 10), 
after which they are integrated using the proposed FRO 
method. 


iw 
e 
Qe) 
. eos Right Finger 


FIGURE 2. FRO-based scores integration using NIST- Set 1. 
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FIGURE 3. Assessment of ROC curves of different unimodal systems with 
score-level Fusion for NIST-Set 1 database. 


In Figure 3, we can observe the Receiver Operating 
Characteristic (ROC) curves for each individual modality as 
well as the combined modalities using the Full 
Reinforcement Operator (FRO) method. These results are 
based on data from the NIST-multimodal database. At a 
False Acceptance Rate (FAR) of 10~*, the Genuine 
Acceptance Rates (GARs), which are calculated as (1-FRR) 
[7], are 74.3% for face matcher C, 68% for face matcher G, 
77.2% for the left fingerprint, and 85.3% for the right 
fingerprint. However, at 10~* FAR operating point, GAR of 
100% is attained with the FRO produced by the products T- 
norm and T-conorm. As indicated in Table V, various 
symmetric sum S-sums produced through many T-norm 
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tules (probabilistic, Yager, and Hamacher [7]) are employed 
for merging scores on the NIST- set1 database. Additionally, 
the study reports the results achieved through combining 
scores using the likelihood ratio (LR) and Support Vector 
Machine (SVM) techniques proposed in [17], both of which 
necessitate learning and training. 


TABLE V: EVALUATION OF FUSION USING DIFFERENT APPROACHES ON 
NIST-SET 1 DATABASE 


Combining approach at FAR = 0.01 % GAR (%) 

Entropy using Hamacher T-norm p = 0.01) [12] 96.7 
Entropy using Frank T-norm p = 0.01) [12] 96.62 
LR-based fusion [17] 99.1 
SVM [17] 98.8 
Adaptive weighted [33] 97.5 
Min rule [34] 78 
Max rule [34] 91.7 
S-sum using min rule [7] 93.2 
S-sum using max rule [7] 93.3 
S-sum with Schweizer & Sklar T-norm with p = 0.9 [7] 99.8 
S-sum with Yager T-norm with p = 1.2 [7] 99.8 
S-sum with Hamacher T-norm [7] 95 
S-sum using probabilistic T-norm [7] 97 
Asym-AO2 using Aczel-Alsina [9] 94 
Asym-AO1 using Aczel-Alsina [9] 96.17 
Asym-AOQ2 using Algebric T-norm [9] 96.90 
Asym-AO1 using Algebric product T-norm [9] 96.5 
Asym-AO2 using Hamacher T-norm [9] 97.55 
Asym-AO1 using Hamacher T-norm [9] 97.50 
Proposed Full Reinforcement Fusion Rule (FRO) 100 


Furthermore, the study showcases results for fusion using the 
Entropy-with-Frank and Entropy-with-Hamacher T-norms 
[7]. Table V clearly illustrates that the FRO, generated by 
both T-norm and T-conorm, outperforms the performance of 
score-level fusion methods previously documented in the 
literature. 


2) Performance FRO on NIST-Set 2 


Figure 4 depicts an architecture that illustrates the complete 
process of a multi-unit person recognition framework based 
on left and right index finger data. The input scores of the 
left and right index fingers for this dataset are normalized 
using the min-max method as in Eqn. 9. The combination 
order is not important because we fused the two matched 
scores here using the commutative property. Using the FRO 
produced by both products (T-norm and T-conorm) on the 
NIST-Set 2 database. Figure 5 shows the ROCs of individual 
modalities and of fused modalities. The left index finger and 
right index finger had GARs of 75.5% and 83.5%, 
respectively, at FAR = 10-*. However, using the same FAR 
operating point and the FRO produced by both products (T- 
norm and T-conorm), a GAR of 93.40% is attained. Table VI 
shows that various S-sums produced using (probabilistic, 
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Yager and Hamacher T-norms) are likewise utilized for 
score-level fusion on the NIST-Set 2 database. For 
comparison, the outcomes of combining scores utilizing the 
LR and SVM learning-based approaches are also provided 
[17]. Table VI shows that the adopted FRO performs better 
than the score-level fusion rules that is currently used in the 
literature. 
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FIGURE 4. FRO based scores integration using NIST-Set 2. 
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FIGURE 5. Performance of FRO score-level Fusion for NIST-Set 2. 


TABLE VI: EVALUATION OF FUSION USING DIFFERENT APPROACHES ON 
NIST-SET 2 DATABASE 


Combining approach at FAR = 0.01 % GAR (%) 

SVM [17] 91.40 
S-sum generated using max rule [17] 90.75 
Adaptive weighted [33] 90.87 
Max rule [34] 90.30 
Min rule [34] 79.60 
LR-based fusion [17] 91.40 
Entropy (using Frank T-norm) [12] 87.42 
Entropy (using Hamacher T-norm) [12] 85.42 
WQAM using cos(s)r with r= 11 [8] 91.60 
S-sum generated by probabilistic T-norm [17] 89 
S-sum generated by Hamacher T-norm [17] 75.5 


S-sum generated by Yager T-norm with p=10.3[17] 90 
S-sum generated by Schweizer & Sklar T-norm [17] 89 


S-sum generated by max rule [17] 90.75 
S-sum generated by min rule [17] 82.5 
Proposed Full Reinforcement Fusion Rule (FRO) 93.40 
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3) Performance of FRO on NIST-Set3 TABLE VII: EVALUATION OF FUSION USING DIFFERENT APPROACHES ON 

NIST-SET 3 DATABASE 
In this section, we provide the results of a multi-algorithm Combining approach at FAR = 0.01 % GAR (%) 
biometric system score fusion technique based on FRO. Asym-AO2 using Aczel-Alsina [9] 76.20 
Figure 6 illustrates the design of a framework for face-based Asym-AO] using Aczel-Alsina [9] 16.23 
multi-algorithm person recognition. The two face-matching Asym-AO2 using Algebric product [9] 76.27 
algorithms (Macher C and Machar G) used by this system. AsyiichGlasee Aleebaeaoduck(S} 76.00 
We specifically experimented on the NIST-face database, ; 
where 3000 subjects have similarity scores. The genuine eee eens ee es 
match-score is 6000 (3000x2), but the impostor match-score Asym-AOI using Hamacher [9] 76.20 
is 17,994,000 (6000x2999). The two match-scores of face Adaptive weighted [33] 85.29 
matcher C and face matcher G were standardized by tanh- Max tule [34] 71.50 
estimators normalization technique as in Eqn. 10. Figure 7 Min rule [34] 73.15 
displays the ROC curves for separate biometric algorithms Sum rule [17] 75.75 
when combined with our suggested score-level fusion Product T-norm [30] 75.85 
a as At the operating point where the FAR is set to Hetashet Aja 130) a866 
10*, we observe GARs of 63.0%, 72.1%, and 94.54% for ; 
the C face matcher, G face matcher, and the multi-algorithm ne Oe ee ioe) 
approach using FRO, respectively. Table VII provides a pesum using Masala [7 ioe 
summary of the authentication rates achieved based on FRO S-sum using Hamacher T-norm [7] 75.75 
using both product T-norm and product T-conorm functions, SVM [17] 77.3 
as well as other fusion techniques proposed previously, such LR-based fusion [17] 77.2 
as SVM, LR, S-sum produced by Hamacher T-norm, S-sum Proposed Full Reinforcement Fusion Rule (FRO) 94.54 
using max rule, Hamacher T-norm, Frank T-norm, sum rule, 
min rule, and max rule. 4) Performance of FRO on XM2VTS Database 


The experimental findings on the XM2VTS databases 
utilizing Lausanne Protocol I and II are presented in this 
section, via modalities (multimodal biometric), extractor 
(multi-algorithm), and classifiers (multi-unit biometric 
system). 

For Lausanne Protocol I, we have employed all 15 possible 
configurations (via modalities). Additionally, we have used 
6 combinations via extractor, and 2 combinations employing 
various classifiers. We also utilized the DS theory, the 
Likelihood ratio-based (LR) combining method, and the sum 
rule for comparisons. The HTER (refer to Eqn. 14) is 
employed to assess and contrast the effectiveness of various 
1 combining methods. 

The outcomes in Table VIII, provide a contrast between 
the suggested combining method (FRO) utilizing product T- 
norm and product T-conorm with DS theory (using a sigmoid 
function), the LR combing rule, and the sum operator. From 
the same Table, it is evident that across the 15 configurations 
Ss Se of the baseline systems (via modalities), the FRO combining 
—< Face Matcher : G rule surpasses the basic sum rule headed by Z-score and 
—+— Face Matcher : C . . . . . : 
on Max-Min normalization in 15 and 12 configurations, 

10% 10° 107 respectively. Nevertheless, the basic sum rule headed by 
Tanh normalization exceeds the FRO combining rule in three 
FIGURE 7. Comparison of ROC curves of individual modality with score- configurations. Likewise, we can see that the projected 
level Fusion for NIST-Set 3. combining method outperforms the DS theory and LR 
approach in 15 and 12 configurations, respectively. 
Moreover, we can determine from Table VIII that over the 
six configurations (via extractor), the proposed FRO 
combining approach surpasses the basic sum rule preceded 
by Z-score, Max-Min, and tanh normalization in 6, 5, and 6 
configurations, respectively. Nevertheless, the LR-based 
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TABLE VIII; ASSESSING THE HTER (%) DIFFERENCES BETWEEN THE FRO-BASED FUSION, DS THEORY-BASED FUSION, THE BASIC SUM RULE, AND 
LR- BASED FUSION FOR THE LAUSANNE PROTOCOL I 


Basic sum rule [31] LR-based fusion 
: : ; ie Proposed DS theory 

SI. Fusion candidates via modalities FRO 31] [31] 
Z-score Min-Max Tanh 

1. (FH, MLP) (LFCC, GMM) 0.595 0.433 0.75 0.862 0.737 1.108 

25 (FH, MLP) (PAC, GMM) 1.018 1.500 1.13 1.161 1.026 1.441 

33 (FH, MLP) (SSC, GMM) 0.531 0.693 0.88 1.072 0.778 1.339 

4. (DCTs,GMM) (LFCC, GMM) 0.433 0.542 0.56 0.492 0.583 0.574 

5. (DCTs,GMM) (PAC, GMM) 1.409 1.421 1.46 1.417 1.476 1.417 

6. (DCTs,GMM) (SSC, GMM) 1.152 1.348 1.14 1.218 1.132 1.201 

Te (DCTb,GMM) (LFCC, GMM) 0.493 0.436 0.53 0.503 0.467 0.499 

8. (DCTb,GMM) (PAC, GMM) 1.399 2.734 1.17 1.093 1.661 1.106 

9. (DCTb,GMM) (SSC, GMM) 0.828 1.385 0.77 0.720 0.733 0.764 

10. (DCTs,MLP) (LFCC, GMM) 0.845 0.589 0.81 0.972 0.728 0.193 

11. (DCTs,MLP) (PAC, GMM) 1.360 2.436 1.19 1.413 0.822 1.982 

12. (DCTs,MLP) (SSC, GMM) 1.343 1.056 1,32 1.594 1.036 1.721 

13. (DCTb,MLP) (LFCC, GMM) 2.428 2.542 2.61 3.278 2.874 2.693 

14. (DCTb,MLP) (PAC, GMM) 2.525 2.875 3.63 4.121 2.623 3.547 

15. (DCTb,MLP) (SSC, GMM) 2.994 1.864 2.83 4,329 2.058 3.722 

prappsed Basic sum rule _ 

SL. Fusion candidates via extractors FRO DS theory LR-based fusion 
Z-score Min-Max Tanh 

1 (FH,MLP) (DCTs,GMM) 0.970 1.283 1.280 1.750 1.743 1.581 

2 (FH,MLP) (DCTb,GMM) 1.624 1.854 1.153 1.735 1.740 1.723 

a. (FH,MLP) (DCTs,MLP) 1.537 1.492 1.510 1.516 1.522 1.573 

4. (FH,MLP) (DCTb,MLP) 1.598 1.609 1.969 1.726 1.711 1.578 

5 (LFCC,GMM) | (SSC,GMM) 1.363 1.013 1.497 0.878 1.141 1.615 

6 (PAC,GMM) (SSC,GMM) 4.202 4.215 4.225 4.244 4.828 3.964 

: : : ips Proposed eens ck 

SL Fusion candidates via classifiers FRO DS theory LR-based fusion 
Z-score Min-Max Tanh 

1. (DCTs,GMM)_ | (DCTs,MLP) 1.654 1.873 2.390 3.359 3.357 1.928 

2. (DCTb,GMM) | (DCTb,MLP) 2.541 3.356 3.061 3.102 3.402 3.203 


Note: The numbers presented in bold fonts indicate the top-performing values within each row [31]. 


TABLE IX: ASSESSING THE HTER (%) DIFFERENCES BETWEEN THE FRO-BASED FUSION, DS THEORY-BASED FUSION, THE BASIC SUM RULE, AND 
LR- BASED FUSION FoR THE LAUSANNE PROTOCOL II 


Basic sum rule [31] LR-based fusion 
SL Fusion candidates via modalities Proposed DS theory [31] 
FRO [31] Z-score Min-Max Tanh 
1. (FH, MLP) (LFCC, GMM) 0.595 0.793 0.688 1.459 1.298 1.245 
2. (FH, MLP) (PAC, GMM) 1.762 1.362 1.144 1.626 1.527 1.674 
3. (FH, MLP) (SSC, GMM) 0.665 0.691 0.981 1.615 1.339 1.339 
4. (DCTb,GMM) (LFCC, GMM) 0.109 0.534 0.133 0.250 0.126 0.342 
5. (DCTb,GMM) (PAC, GMM) 0.141 1.212 0.175 0.504 1.220 1.487 
6. (DCTb,GMM) (SSC, GMM) 0.342 1.258 0.177 0.445 1.223 1.207 
Basic sum rule 
SL. Fusion candidates via extractors Proposed DS theory LR-based fusion 
FRO Z-score Min-Max Tanh 
1. (FH, MLP) (DCTs,GMM) 0.970 1.163 0.962 1.486 1.485 1.481 
' (FH, MLP) (DCTb,GMM) 1.624 1.736 0.828 1.105 1.101 1.420 
3. (FH, MLP) (DCTs,MLP) 1.437 1.455 2.891 2.978 3.033 2.762 


Note: The numbers presented in bold fonts indicate the top-performing values within each row [31]. 


fusion surpasses the FRO based-fusion in two combinations. strategies. For Lausanne Protocol I, we have employed all 
Also, the proposed FRO based-fusion outperforms the DS six possible configurations of face and speech classifiers (via 
theory in 5 configurations. Finally, we can see from Table modalities). Additionally, we have used three combinations 
VIII that over the two combinations (via classifiers), the via extractor. We also utilized the DS theory, Likelihood 
proposed fusion method outperforms all other score fusion ratio technique, and sum rule for the purpose of making 
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comparisons. The HTER is employed to assess and contrast 
the effectiveness of various fusion methods (refer to Eqn. 
14). The outcomes in Table IX provide an assessment 
between the adopted fusion technique (FRO) using product 
T-norm and product T-conorm with DS theory (using 
sigmoid function), the basic sum rule, and the LR fusion rule. 
From the same Table, it is evident that across the 6 
configurations (via modalities), the FRO outperforms the 
basic sum rule preceded by tanh and Min-Max normalization 
in all configuration. But the basic sum rule preceded by Z- 
score normalization surpasses the FRO based-fusion in 2 
configurations. Moreover, we can discern from Table IX that 
over the 3 configurations (via extractor), the proposed FRO 
based fusion outperforms the simple sum rule preceded by 
Z-score, Min-Max, DS theory, and LR-based fusion in all 
configurations. However, the basic sum rule preceded by 
tanh normalization surpasses the FRO based- fusion in one 
configuration ((FH, MLP), (DCTb, GMM)). 


VI. CONCLUSION 

In this paper, we introduce an innovative score fusion 
method for a system that uses multiple biometrics traits, that 
relies on the complete reinforcement operator (FRO). This 
FRO does not necessitate any form of learning or training, 
which simplifies our system, enhances its efficiency, and 
reduces computational costs. 

Experiments were conducted using three distinct subsets 
of the NIST-BSSR1 dataset (set 1, set 2, set 3) in addition to 
the XM2VTS database. The outcomes obtained from NIST- 
multimodal biometric tests demonstrated exceptional 
performance of our proposed score-level fusion method, 
which utilizes product T-norm and product T-conorm. 
Specifically, it attains a high Genuine Acceptance Rate 
(GAR) of 100% at a False Acceptance Rate (FAR) of 0.01%. 
This performance surpasses that of individual modalities and 
outperforms various conventional score-level fusion 
methods, including S-sum, Entropy, Asym-AOs, Yager T- 
norm, Product T-norm, Sum, LR-based method, and SVM. 
In the context of NIST fingerprint data, our suggested score- 
level fusion technique achieves superior performance, with a 
GAR of 92.8% at a FAR of 0.01%, surpassing the 
performance of all previously employed methods. For the 
last subset of NIST-BSSR1 (NIST-face database), the 
proposed score-level fusion technique achieves superior 
performance, with a GAR of 92.8% at a FAR of 0.01%, 
surpassing the performance of all previously used 
approaches. 

The test of the approach on the publicly available scores 
of the XM2VTS Benchmark gives better performance 
(HTER) compared to various conventional existing 
approaches, including DS theory, LR-based, sum rule using 
tanh normalization, sum rule using min-max normalization, 
and sum rule preceded by Z-score. 

To summarize, our experimental findings unequivocally 
demonstrate the superiority of the proposed framework when 
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compared to existing methods like DS theory and SVM. 
Nevertheless, a more extensive evaluation, including mobile 
authentication databases, is imperative and will be pursued 
in future research. Additionally, we aim to assess the model’s 
resistance against counterfeiting attempts in our upcoming 
endeavors. 
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